FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization Approach for Deep Neural Networks

The design of machine learning systems often requires trading off different objectives, for example, prediction error and energy consumption deep neural networks (DNNs). Typically, no single performs well in all objectives; therefore, finding Pareto-optimal designs is interest. search involves evaluating an iterative process, the measurements are used to evaluate acquisition function that guides process. However, measuring objectives incurs costs. For cost DNNs orders magnitude higher than a pre-trained DNN as it re-training DNN. Current state-of-the-art methods do not consider this difference objective evaluation cost, potentially incurring expensive evaluations functions optimization In paper, we develop novel decoupled cost-aware multi-objective algorithm, which call Flexible Multi-Objective Bayesian Optimization (FlexiBO) address issue. each design, FlexiBO selects with relative gain by weighting improvement hypervolume Pareto region measurement objective. This strategy, balances expense collecting new information knowledge gained through evaluations, preventing from performing little gain. We on seven image recognition, natural language processing (NLP), speech-to-text translation. Our results indicate that, given same total experimental budget, discovers 4.8% 12.4% lower best method optimization.